Web Survey Bibliography
Title Web Probing for Question Evaluation: The Effects of Probe Placement
Author Fowler, S.; Willis, G. B.; Moser, R. P.; Townsend, R. L. M.; Maitland, A.; Sun, H.; Berrigan, D.
Year 2016
Access date 09.06.2016
Abstract
Though the majority of cognitive interviewing is conducted in a laboratory environment by specialized interviewers, an alternative procedure,embedded probing, was developed decades ago (Converse & Presser, 1986; Schuman, 1966). Embedded probing involves the evaluation of select items through the administration of cognitive probes within a field survey environment, relying on much larger samples than are cost-effective with standard cognitive interviewing. Embedded probing has recently resurfaced in the form of web probing, mainly prompted by the increasing availability of internet-based platforms that provide fast, inexpensive access to respondents (Behr, Braun, Kaczmirek, & Bandilla,
2013; Murphy, Keating, and Edgar, 2013). However, little guidance exists concerning specific practices. In particular, as for standard cognitive interviewing, it is unclear whether probes should be administered concurrently with the target item, or retrospectively at the end of the interview (i.e. debriefing). This study examined web-based probing in the context of questionnaire pretesting and evaluation, relying on respondents from Amazon’s Mechanical Turk (mTurk), an online labor crowdsourcing platform. A questionnaire segment on walking behavior and attitudes, selected from the National Health Interview Survey (NHIS), was presented to 1,444 mTurk respondents. Probes were embedded for 4 targeted items. For example, respondents were asked: “When answering the question -- Where you live, are there roads, sidewalks, paths, or trails where you can walk? – Please say more about what you were thinking about...”, then typed their response to the probe into an open text box. As an experimental manipulation, approximately half of respondents answered the probes concurrently, and the remainder responded retrospectively. We will present the results of an analysis that compares the responses obtained as a function of probe placement, by assessing quantitative metrics such as number of words entered, as well as qualitative measures, including thematic coding and rater judgment of usefulness of the information obtained.
2013; Murphy, Keating, and Edgar, 2013). However, little guidance exists concerning specific practices. In particular, as for standard cognitive interviewing, it is unclear whether probes should be administered concurrently with the target item, or retrospectively at the end of the interview (i.e. debriefing). This study examined web-based probing in the context of questionnaire pretesting and evaluation, relying on respondents from Amazon’s Mechanical Turk (mTurk), an online labor crowdsourcing platform. A questionnaire segment on walking behavior and attitudes, selected from the National Health Interview Survey (NHIS), was presented to 1,444 mTurk respondents. Probes were embedded for 4 targeted items. For example, respondents were asked: “When answering the question -- Where you live, are there roads, sidewalks, paths, or trails where you can walk? – Please say more about what you were thinking about...”, then typed their response to the probe into an open text box. As an experimental manipulation, approximately half of respondents answered the probes concurrently, and the remainder responded retrospectively. We will present the results of an analysis that compares the responses obtained as a function of probe placement, by assessing quantitative metrics such as number of words entered, as well as qualitative measures, including thematic coding and rater judgment of usefulness of the information obtained.
Access/Direct link Conference Homepage (abstract)
Year of publication2016
Bibliographic typeConferences, workshops, tutorials, presentations
Web survey bibliography (4086)
- Respondent Processing of Rating Scales and the Scale Direction Effect ; 2016; Caporaso, A.
- The Effects of Pictorial vs. Verbal Examples on Survey Responses ; 2016; Sun, H.; Bertling, J.; Almonte, D.
- Evaluating Grid Questions for 4th Graders; 2016; Maitland, A.
- Mixing Modes: Challenges (and Tradeoffs) of Adapting a Mailed Paper Survey to the Web ; 2016; Wilkinson-Flicker, S.; McPhee, C. B.; Medway, R.; Kaiser, A.; Cutts, K.
- An Examination of How Survey Mode Affect Eligibility, Response and Health Condition Reporting Rates...; 2016; Stern, M. J.; Ghandour, R.
- Investigating Measurement Error through Survey Question Placement ; 2016; Wilson, A.; Wine, J.; Janson, N.; Conzelmann, J.; Peytcheva, E.
- Instructions in Self-administered Survey Questions: Do They Improve Data Quality or Just Make the Questionnaire...; 2016; Redline, C. D.; Zukerberg, A.; Owens, C.; Ho, A.
- Usability Testing within Agile Process; 2016; Holland, T.
- Exploring Why Web Surveys Take Longer to Complete on Smartphones than PCs: Findings from a Within-subjects...; 2016; Antoun, C.; Cernat, A.
- Making Mobile Web Surveys Accessible; 2016; Malakhoff, L.
- Association of Eye Tracking with Other Usability Metrics ; 2016; Olmsted, E. L.
- Cognitive Probing Methods in Usability Testing – Pros and Cons; 2016; Nichols, E. M.
- Grids and Online Surveys: Do More Complex Grids Induce Survey Satisficing? Evidence from the Gallup...; 2016; Wang, Me.; McCutcheon, A. L.
- Assessing the Accuracy of 51 Nonprobability Online Panels and River Samples: A Study of the Advertising...; 2016; Yang,Y.;Callegaro,M.;Yang,Y.;Callegaro,M.;Chin,K.;Yang,Y.;Villar,A.;Callegaro, M.; Chin, K.; Krosnick...
- Calculating Standard Errors for Nonprobability Samples when Matching to Probability Samples ; 2016; Lee, Ad.; ZuWallack, R. S.
- Communicating Data Use and Privacy: In-person versus Web based methods for message testing ; 2016; Clark Fobia, A.; Hunter Childs, J. E.
- User Experience and Eye-tracking: Results to Optimize Completion of a Web Survey and Website Design ; 2016; Walton, L.; Ricci, K.; Libman Barry, A.; Eiginger, C.; Christian, L. M.
- Estimated-control Calibrated Estimates from Nonprobability Surveys; 2016; Dever, J. A.
- Decomposing Selection Effects in Non-probability Samples ; 2016; Mercer, A. W.; Keeter, S.; Kreuter, F.
- The Effect of Emphasizing the Web Option in a Mixed-mode Establishment Survey ; 2016; O'Brien, J.; Rajapaksa, S.; Schafer, B.; Langetieg, P.
- A Multi-phase Exploration Into Web-based Panel Respondents: Assessing Differences in Recruitment, Respondents...; 2016; Redlawsk, D.; Rogers, K.; Borie-Holtz, D.
- Effect of Clarifying Instructions on Response to Numerical Open-ended Questions in Self-administered...; 2016; Kumar Chaudhary, A.; Israel, G. D.
- Exploring the Feasibility of Using Facebook for Surveying Special Interest Populations ; 2016; Lee, C.; Jang, S.
- National Estimates of Sexual Minority Women Alcohol Use through Web Based Respondent Driven Sampling...; 2016; Farrell Middleton, D.; Iachan, R.; Freedner-Maguire, N.; Trocki, K.; Evans, C.
- Bringing Fair Market Rent Surveys into the 21st Century – Evaluating the Effectiveness of MSG...; 2016; Dayton, J.; Brassell, T.; Cooper, V.; Dion, R.; Williams, R.
- Measuring Survey Behavior of Smartphone Users; 2016; Luks, S.; Phillips, R.
- Practical Considerations for Using Vignettes to Evaluate Survey Items ; 2016; Steiger, D. M.; Williams, Do.; Edwards, W. S.; Cantor, D.; Truman, J.
- Using Web Panels to Quantify the Qualitative: The National Center for Health Statistics Research and...; 2016; Scanlon, P. J.
- Impact of Field Period Length in the Estimates of Sexual Victimization in a Web-based Survey of College...; 2016; Berzofsky, M.; Peterson, K.; Shook-Sa, B. E.; Lindquist, C.; Krebs, C.
- Longitudinal Online Ego-centric Social Network Data Collection with EgoWeb 2.0 ; 2016; Amin, A.; Kennedy, D.
- Influences on Item Response Times in a Multinational Web Survey ; 2016; Phillips, B. T.; Kolenikov, S.; Howard Ecklund, E.; Ackermann, A.; Brulia, A.
- QR Codes for Survey Access: Is It Worth It?; 2016; Allen, L.; Marlar, J.
- An Exploration of the Relationship between Usability Testing and Data Verification ; 2016; Langer Tesfaye, C.; Kurmlavage, V.
- Beyond the Survey: Improving Data Insights and User Experience with Mobile Devices ; 2016; Graham, P.; Lew, G.
- User Experience Considerations for Contextual Product Surveys on Smartphones ; 2016; Sedley, A.; Mueller, H.
- The Differential Effect of Mobile-friendly Surveys on Data Quality; 2016; Horwitz, R.
- Embedding Survey Questions within Non-research Mobile Apps: A Method for Collecting High-quality Data...; 2016; Bapna, V.; Antoun, C.
- Does Changing Monetary Incentive Schemes in Panel Studies Affect Cooperation? A Quasi-experiment on...; 2016; Schaurer, I.; Bosnjak, M.
- Survey Mode and Mail Method: A Practical Experiment in Survey Fielding for a Multi-round Survey ; 2016; Sullivan, B. D.; Duda, N.; Bogen, K.; Clusen, N. A.; Wakar, B.; Zhou, H.
- Web Probing for Question Evaluation: The Effects of Probe Placement ; 2016; Fowler, S.; Willis, G. B.; Moser, R. P.; Townsend, R. L. M.; Maitland, A.; Sun, H.; Berrigan, D.
- Early-bird Incentives: Results From an Experiment to Determine Response Rate and Cost Effects ; 2016; De Santis, J.; Callahan, R.; Marsh, S.; Perez-Johnson, I.
- Using Cash Incentives to Help Recruitment in a Probability Based Web Panel: The Effects on Sign Up Rates...; 2016; Krieger, U.
- Assessing Changes in Coverage Bias of Web Surveys a s Internet Access Increases in the United States...; 2016; Sterrett, D.; Malato, D.; Benz, J.; Tompson, T.; English, N.
- Timing is Everything: Discretely Discouraging Mobile Survey Response through the Timing of Email Contacts...; 2016; Richards, A.; C.; Shook-Sa, B. E.; C.; Berzofsky, M.; Smith, A. C.
- Dynamic Instructions in Check-All-That-Apply Questions ; 2016; Kunz, T.; Fuchs, M.
- Patterns of Unit and Item Nonresponse in a Multinational Web Survey ; 2016; Ackermann, A.; Howard Ecklund, E.; Phillips, B. T.; Brulia, A.
- Debunking Myths About the Quality of Industry and O ccupation Data Collected Through Self-administered...; 2016; Hurwitz, F. I.; Stein, J.; Skaff, A. L.
- Desktops, Tablets and Phones, Oh My! Device Prefere nce for Web Based Surveys ; 2016; Schy, S.; Ghirardelli, A.; Morrison, H.
- Assessing Potential Bias in Respondent-driven Incident Based Data from a Web Survey of College Students...; 2016; Peterson, K.; Berzofsky, M.; Shook-Sa, B. E.; Krebs, C.; Lindquist, C.
- Making Connections on the Internet: Online Survey Panel Communications ; 2016; Libman Barry, A.; Eiginger, C.; Walton, L.; Ricci, K.